Information processing apparatus, method, and storage medium storing program

ABSTRACT

A first still image included in a still image file and a second still image extracted from a moving image included in a moving image file are evaluated as layout target on a template. The layout target on the template is specified out of the first still image and the second still image in accordance with the evaluation, and the still image file of the layout target is laid out on the template. The first still image and the second still image are evaluated such that the possibility that the second still image is specified becomes higher than that for the first still image.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information processing apparatus forperforming layout processing of a file, a method, and a storage mediumstoring a program.

2. Description of the Related Art

There is recently a product that automatically lays out a plurality ofstill images on a template for a photo album using a photo albumcreation application, thereby automatically generating a photo album.Additionally, as a technique of extracting a still image from a movingimage, there is a technique of specifying a plurality of frame groups ofa moving image based on a frame interval according to a user instructionand extracting an output target frame based on the analysis result of aplurality of frames included in each frame group (Japanese PatentLaid-Open No. 2013-26935).

Even in a case in which a user wants to create a photo album on whichboth a still image and a moving image are laid out, if the applicationfor automatically creating a photo album does not handle a moving imageas an image for a photo album, an image included in the moving image isnot laid out by the application.

SUMMARY OF THE INVENTION

An aspect of the present invention is to eliminate the above-mentionedproblems with the conventional technology. The present inventionprovides an information processing apparatus that appropriately performslayout processing including an image extracted from a moving image, amethod, and a storage medium storing a program.

The present invention in one aspect provides a method, comprising:evaluating a first still image included in a still image file and asecond still image extracted from a moving image included in a movingimage file as layout targets on a template; specifying a still image ofthe layout target on the template out of the first still image and thesecond still image in accordance with the evaluation; and laying out thespecified still image of the layout target on the template, wherein thefirst still image and the second still image are evaluated such that apossibility that the second still image is specified as the layouttarget on the template becomes higher than that for the first stillimage.

According to the present invention, it is possible to appropriatelyperform layout processing including an image extracted from a movingimage.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the arrangement of a system includingan information processing apparatus;

FIG. 2 is a block diagram showing the hardware arrangement of a PC;

FIG. 3 is a block diagram showing the software configuration of the PC;

FIGS. 4A to 4C are views showing the screens of an application;

FIG. 5 is a flowchart showing automatic photo album creation processing;

FIG. 6 is a flowchart showing automatic photo album creation processing;

FIG. 7 is a flowchart showing still image classification processing;

FIGS. 8A to 8C are flowcharts showing processing of extracting a stillimage from a moving image;

FIG. 9 is a flowchart showing automatic photo album creation processing;

FIGS. 10A and 10B are views showing structures of photo albums;

FIGS. 11A to 11C are views showing layout templates;

FIGS. 12A and 12B are views for explaining a method of managing slots ofa layout template;

FIGS. 13A to 13E are views showing the relationship of layout betweenslots and still images;

FIG. 14 is a view showing the scores of still images;

FIG. 15 is a view showing the position information of slots;

FIGS. 16A and 16B are flowcharts showing processing of acquiring a stillimage from a moving image;

FIGS. 17A and 17B are views showing association between still images andmoving images and classification of still images; and

FIG. 18 is a view showing frames of a moving image.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will now be describedhereinafter in detail, with reference to the accompanying drawings. Itis to be understood that the following embodiments are not intended tolimit the claims of the present invention, and that not all of thecombinations of the aspects that are described according to thefollowing embodiments are necessarily required with respect to the meansto solve the problems according to the present invention.

First Embodiment

FIG. 1 is a block diagram showing an example of the arrangement of aphoto album creation/order system formed from an information processingapparatus. Referring to FIG. 1, each of information processingapparatuses 1 and 2 is formed from, for example, a general personalcomputer or server (to be sometimes abbreviated as a PC hereinafter).Each of the PCs 1 and 2 is formed from hardware to be described laterwith reference to FIG. 2, in which an operating system (to be sometimesabbreviated as an OS hereinafter) equivalent to Windows® 8 or Windows®Server 2012 is installed as an OS. The PCs 1 and 2 are connected tonetworks 4 and 8 formed from Ethernet®, respectively.

A photo album creation application 80 (to be sometimes abbreviated as anapplication hereinafter) is formed from a file (*.EXE) in a formatexecutable on, for example, Windows®. A web browser application 143 (tobe sometimes abbreviated as a web browser hereinafter) is a browser usedwhen using WWW (World Wide Web). The network 4 is a home network for ageneral household, which is constructed in a home where the user (forexample, a customer) who uses the PC 1 inhabits. The network 8 is anoffice network constructed in the ABC company.

The PC 2 connected to the network 8 includes a web server 9 having thefunction of a web server, and provides the web site of the ABC companyvia the Internet. A shopping site 11 is a shopping site on which the ABCcompany provides a service for users. The user can order a photo albumusing the shopping site 11.

FIG. 2 is a block diagram showing an example of the hardware arrangementof the PC. Each of the PCs 1 and 2 includes hardware as shown in FIG. 2.As shown in FIG. 2, each of the PCs 1 and 2 includes a RAM 201 that is arandom access memory unit, an HDD 202 that is a hard disk drive unit, aKBD 203 that is a keyboard unit, and a CPU 204. Each of the PCs 1 and 2further includes an LCD 205 that is a display, and an NB 207 that is anetwork board. Each of the PCs 1 and 2 also includes a bus 206 thatcommunicably connects at least the blocks shown in FIG. 2 to each other.Note that the HDD 202 is, for example, the storage unit of the PC 1 or2. The storage unit may be a portable CD-ROM, a fixed internal ROM, orthe like. The KBD 203 is, for example, the input unit of the PC 1 or 2.The CPU 204 is, for example, the control unit of the PC 1 or 2. The LCD205 is, for example, the display unit of the PC 1 or 2. The NB 207 is,for example, the communication control unit of the PC 1 or 2.

Applications such as the application 80 and the web browser 143 andmodules (software) shown in FIG. 3 are stored in, for example, the HDD202, read out to the RAM 201 as needed, and executed by the CPU 204. TheCPU 204 thus implements the functions of applications such as theapplication 80 and the web browser 143 and modules (software) shown inFIG. 3.

FIG. 3 is a block diagram showing an example of the module configurationof the software of the PC. Referring to FIG. 3, a module 92 is anEthernet® control stack for controlling Ethernet®. A module 91 is an IPnetwork control stack for controlling an IP network. A module 90 is aWSD control stack for controlling a WSD (Web Service on Devices) thatprovides a mechanism to search for a device on a network. A module 88 isa PnP-X control stack for controlling plug and play of a network. Notethat PnP-X stands for Plug and Play Extensions that is installed in theWindows® 8 OS as a series of standard extension functions, and providesa support for a network connected device. A module 85 is a device drivergroup and includes standard drivers 87 bundled in the OS and IHV drivers86 provided by Independent Hardware Vendor (IHV).

A module 84 is an application/DDI interface and includes ApplicationProgramming Interface (API) and Device Driver Interface (DDI). Themodule 80 is, for example, a photo album creation application, and themodule 143 is, for example, a web browser application. A module 82 is anapplication group and includes the application 80 and the web browser143.

FIGS. 4A to 4C are views showing examples of the screens of theapplication. FIG. 4A shows an example of the main screen of theapplication 80. FIG. 4B shows an example of the preview/editing screenof the application 80. FIG. 4C shows an example of a photo album orderscreen displayed on the web browser 143.

Referring to FIG. 4A, a screen 401 is main screen of the application 80.A selection portion 402 is a photo album type selection portion, whichcan accept selection of one of type-K and type-M as the type of a photoalbum to be created. A button 403 is a photo album skeleton buttonconfigured to accept selection of a photo album of type-K. A photo albumof type-K is, for example, a photo album as follows.

Shape: 15 cm×15 cm square

Number of pages: 20

Paper type: satin

Binding: half fold

Cover: soft

A button 404 is a photo album skeleton button configured to acceptselection of a photo album of type-M. A photo album of type-M is, forexample, a photo album as follows.

Shape: A4 (210 mm×297 mm) Portrait

Number of pages: 50

Paper type: glossy

Binding: flat

Cover: hard

A mark 405 is a photo album selection state indicating mark. The mark405 is displayed on the photo album selection button associated with theselected photo album. FIG. 4A shows a state in which the button 403 ispressed, and a photo album of type-K is selected. A button 412 is aphoto album creation button. Upon detecting the pressing of the button412, the application 80 starts creating a photo album. A selectionportion 406 is a moving image file processing method selection portion,which accepts selection of one of auto, time, scene, and theme by aradio button as a processing method of extracting a still image from amoving image.

A button 407 is an auto selection button. As a processing method ofextracting a still image from a moving image, a method of executingautomatic processing shown in FIG. 5 (to be described later) correspondsto the button 407. The processing is executed when the button 407 ispressed. A button 408 is a time selection button. As a processing methodof extracting a still image from a moving image, a method of executingprocessing based on a shooting date/time as shown in FIG. 8A (to bedescribed later) corresponds to the button 408. The processing isexecuted when the button 408 is pressed. A button 409 is a sceneselection button. As a processing method of extracting a still imagefrom a moving image, a method of executing screen determinationprocessing based on a color and histogram as shown in FIG. 8B (to bedescribed later) corresponds to the button 409. The processing isexecuted when the button 409 is pressed. A button 410 is a themeselection button. As a processing method of extracting a still imagefrom a moving image, a method of executing theme determinationprocessing based on an object as shown in FIG. 8C (to be describedlater) corresponds to the button 410. The processing is executed whenthe button 410 is pressed.

A button 411 is a file selection button, which is enabled when the sceneselection button 409 or theme selection button 410 is selected. Upondetecting the pressing of the button 411 by the user, a file selectionscreen (not shown) is displayed, and the user is allowed to select astill image file. The application 80 temporarily saves a still imagefile selected by the user via the file selection screen in the workmemory in the RAM 201 used by the application 80. Upon detecting thepressing of the photo album creation button 412 by the user, theapplication 80 executes the above-described scene determinationprocessing or theme determination processing based on the still imagefile temporarily saved in the work memory. A frame extraction timeinterval designation portion 413 can accept designation of a timeinterval to extract a still image from a moving image on, for example, asecond basis from the user. Note that the default value is set to, forexample, 1 sec.

Referring to FIG. 4B, a screen 415 is the preview/editing screen of theapplication 80. A preview display portion 418 is a preview displayportion. The application 80 displays, in the preview display portion418, a preview of a double spread of the photo album that is beingcreated. A double spread selection portion 416 can accept, from theuser, selection of a double spread to be displayed in the previewdisplay portion 418. A double spread selection state indicating mark 417is displayed on the double spread selected by the user. FIG. 4B shows astate in which a double spread including pages 7 and 8 is selected.

A purchased merchandise addition button 419 can add a photo album to ashopping cart (not shown) as merchandize when the user purchases it.Upon detecting the pressing of the purchased merchandise addition button419 by the user, the application 80 adds the photo album displayed inthe preview display portion 418 to the shopping cart. The user canpurchase the photo album added to the shopping cart via an online ordersite shown in FIG. 4C, which is associated with the shopping cart, and asettlement site (not shown) that cooperates with the online order site.Note that the online order site, the settlement site, and the shoppingcart are the same as a general online shopping site and a settlementsite and a shopping cart cooperating with it, and a description thereofwill be omitted.

Referring to FIG. 4C, a URL setting portion 421 accepts designation ofthe URL of a web site (page) that the user wants to display on the webbrowser 143. The URL of the web site (page) displayed on the web browser143 is also displayed in the URL setting portion 421. A screen 422 is aphoto album order screen displayed on the web browser 143. FIG. 4C showsan example in which the user orders a photo album of order number 4649.An order information display portion 423 displays information aboutorder of the photo album. FIG. 4C shows an example as follows.

Order number: 4649

Photo album type: type-K

Price: ¥100,000

The user presses an order button 424, and inputs necessary informationsuch as a name, an address, a telephone number, and a deliverydestination in accordance with a screen displayed after that. Whensettlement is completed in the settlement site, the user can order thephoto album created by the application 80 and purchase the album.

FIGS. 10A and 10B are views showing examples of the structures of photoalbums. FIG. 10A shows an example of the structure of a photo album oftype-K, and FIG. 10B shows an example of the structure of a photo albumof type-M.

The photo album shown in FIG. 10A includes a front cover 1001, a backcover 1004, a double spread 1002 including pages 1 and 2 of the bookbody, and a double spread 1003 including pages 19 and 20 of the bookbody. In the photo album of type-K, the first page is the start page,and the 20th page is the last page concerning the book body.

The photo album shown in FIG. 10B is a photo album of type-M, whichincludes a front cover 1011, a back cover 1014, a double spread 1012including pages 1 and 2 of the book body, and a double spread 1013including pages 49 and 50 of the book body. In the photo album oftype-M, the first page is the start page, and the 50th page is the lastpage concerning the book body.

FIGS. 11A to 11C are views showing examples of layout templates (to bealso referred to as templates hereinafter). FIG. 11A shows an example ofa time base layout template of a photo album of type-K. FIG. 11B showsan example of a scene base layout template of a photo album of type-K.FIG. 11C shows an example of a theme base layout template of a photoalbum of type-K.

Referring to FIG. 11A, a template 1100 is, for example, the layouttemplate of a double spread including pages 1 and 2 of the book body outof the time base layout template of a photo album of type-K. A template1101 is the layout template of page 1 of the book body, and a template1102 is the layout template of page 2 of the book body. Slots 1103,1104, 1105, and 1106 are slots laid out on page 1 of the book body. Theapplication 80 lays out still images in these slots.

The slots 1103, 1104, 1105, and 1106 belong to the region of group 1. Acode K-A-G1-P1 added to the template 1101 represents the attribute ofthis template. K represents a photo album of type-K, A represents timebase, G1 represents the region of group 1, and P1 represents page 1 ofthe book body. Codes K-A-G1-1, K-A-G1-2, K-A-G1-3, and K-A-G1-4 added tothe slots 1103, 1104, 1105, and 1106 represent the attributes of theseslots, respectively. K-A-G1 indicates the same contents as describedabove. The last number indicates the slot number. The slot number isassigned on a group basis.

Slots 1107, 1108, 1109, and 1110 are slots laid out on page 2 of thebook body. The application 80 lays out still images in these slots. Theslots 1107, 1108, 1109, and 1110 belong to the region of group 2. A codeK-A-G2-P2 added to the template 1102 represents the attribute of thistemplate. K-A indicates the same contents as described above. G2represents the region of group 2, and P2 represents page 2 of the bookbody. Codes K-A-G2-1, K-A-G2-2, K-A-G2-3, and K-A-G2-4 added to theslots 1107, 1108, 1109, and 1110 represent the attributes of theseslots, respectively. K-A-G2 indicates the same contents as describedabove. The last number indicates the slot number. The slot number isassigned on a group basis. FIG. 11A shows an example in which regionsare put into groups on a page basis, and all slots in a group (a page onone side) belong to the region of the same group.

Referring to FIG. 11B, a template 1120 is, for example, the layouttemplate of a double spread including pages 7 and 8 of the book body outof the scene base layout template of a photo album of type-K. A template1121 is the layout template of page 7 of the book body, and a template1122 is the layout template of page 8 of the book body. A slot 1124 is aslot laid out on page 7 of the book body, a slot 1126 is a slot laid outon page 8 of the book body, and slots 1123 and 1125 are slots laid outacross pages 7 and 8 of the book body. The application 80 lays out stillimages in these slots.

The slots 1123, 1124, 1125, and 1126 belong to the region of group 4. Acode K-B-G4-P7 added to the template 1121 and a code K-B-G4-P8 added tothe template 1122 represent the attributes of these templates,respectively. K represents a photo album of type-K, B represents scenebase, G4 represents the region of group 4, P7 represents page 7 of thebook body, and P8 represents page 8 of the book body. Codes K-B-G4-1,K-B-G4-2, K-B-G4-3, and K-B-G4-4 added to the slots 1123, 1124, 1125,and 1126 represent the attributes of these slots, respectively. K-B-G4indicates the same contents as described above. The last numberindicates the slot number. The slot number is assigned on a group basis.FIG. 11B shows an example in which regions are put into groups on adouble spread basis, and all slots in a group (double spread) belong tothe region of the same group.

Referring to FIG. 11C, a template 1140 is, for example, the layouttemplate of a double spread including pages 5 and 6 of the book body outof the theme base layout template of a photo album of type-K. A template1141 is the layout template of page 5 of the book body, and a template1142 is the layout template of page 6 of the book body. Slots 1143,1144, 1145, 1146, and 1147 are slots laid out on page 5 of the bookbody. The application 80 lays out still images in these slots.

The slots 1143, 1144, 1145, 1146, and 1147 belong to the region of group5. A code K-C-G5-P5 added to the template 1141 represents the attributeof this template. K represents a photo album of type-K, C representstheme base, G5 represents the region of group 5, and P5 represents page5 of the book body. Codes K-C-G5-1, K-C-G5-2, K-C-G5-3, K-C-G5-4, andK-C-G5-5 added to the slots 1143, 1144, 1145, 1146, and 1147 representthe attributes of these slots, respectively. K-C-G5 indicates the samecontents as described above. The last number indicates the slot number.The slot number is assigned on a group basis. Slots 1148, 1149, 1150,and 1151 are slots laid out on page 6 of the book body. The application80 lays out still images in these slots.

The slots 1148, 1149, 1150, and 1151 belong to the region of group 6. Acode K-C-G6-P6 added to the template 1142 represents the attribute ofthis template. K-C indicates the same contents as described above. G6represents the region of group 6, and P6 represents page 6 of the bookbody. Codes K-C-G6-1, K-C-G6-2, K-C-G6-3, and K-C-G6-4 added to theslots 1148, 1149, 1150, and 1151 represent the attributes of theseslots, respectively. K-C-G6 indicates the same contents as describedabove. The last number indicates the slot number. The slot number isassigned on a group basis. FIG. 11C shows an example in which regionsare put into groups on a page basis, and all slots in a group (a page onone side) belong to the region of the same group.

FIGS. 12A and 12B are views for explaining a layout template and slotmanagement. FIG. 12A shows an example of the scene base layout templateof a photo album of type-K, and FIG. 12B shows an example of the slotmanagement method. FIG. 12A shows a template as an improved version ofthe layout template shown in FIG. 11B, and illustrates an example inwhich still images and still images extracted from a moving image arelaid out in appropriate slots.

Referring to FIG. 12A, a template 1200 is the layout template of adouble spread including pages 15 and 16 of the book body out of thescene base layout template of a photo album of type-K. A template 1201is the layout template of page 15 of the book body, and a template 1202is the layout template of page 16 of the book body. Slots 1205 and 1206are slots laid out on page 15 of the book body, and slots 1204, 1208,and 1209 are slots laid out on page 16 of the book body. Slots 1203 and1207 are slots laid out across pages 15 and 16 of the book body. Theapplication 80 lays out still images in these slots.

The slots 1203, 1204, 1205, 1206, 1207, 1208, and 1209 belong to theregion of group 8. A code K-B-G8-P15 added to the template 1201 and acode K-B-G8-P16 added to the template 1202 represent the attributes ofthese templates, respectively. K represents a photo album of type-K, Brepresents scene base, G8 represents the region of group 8, P15represents page 15 of the book body, and P16 represents page 16 of thebook body.

Codes (K-B-G8-1-S, and the like) added to the slots 1203, 1204, 1205,1206, 1207, 1208, and 1209 represent the attributes of these slots,respectively. K-B-G8 indicates the same contents as described above. Thenext number indicates the slot number. The last letter indicates theattribute of an image to be laid out. S indicates an attributerepresenting that the slot is used for a general still image such as aphoto shot by a digital still camera or the like and saved. D indicatesan attribute representing that the slot is used for a still imageextracted from a moving image by performing frame analysis or the like.For example, a code K-B-G8-1-S is added to the slot 1203. Since theimage attribute is S, an image suitable for layout in this slot is ageneral still image. Additionally, for example, a code K-B-G8-3-D isadded to the slot 1205. Since the image attribute is D, an imagesuitable for layout in this slot is a still image extracted from amoving image. The slot number is assigned on a group basis.

FIG. 12A shows an example in which regions are put into groups on adouble spread basis, and all slots in a group (double spread) belong tothe region of the same group. Note that if a general still imagesuitable for layout in the slot 1203 suitable for a general still imagedoes not exist, a still image extracted from a moving image may be laidout in this slot. In addition, if a still image extracted from a movingimage suitable for layout in the slot 1205 suitable for a still imageextracted from a moving image does not exist, a general still image maybe laid out in this slot.

In some cases, for example, a still image that is slightly out of focusand less appropriate for a photo album exists among a plurality of stillimages extracted from a moving image, and there exists a general stillimage whose quality is higher than that of the still image slightly outof focus. In this case as well, the still image slightly out of focuscan effectively be laid out in a slot of attribute D suitable for astill image extracted from a moving image as one of continuous stillimages extracted from a moving image. That is, the plurality ofcontinuous still images extracted from the moving image are laid out inthe slots of attribute D suitable for a still image as a series of stillimages. This makes it possible to automatically create a dynamic photoalbum exploiting the feature of the layout template without a cumbersomeoperation of the user. It is also possible to provide a photo albumcreation application having a function of automatically generating anattractive photo album.

FIG. 12B is a view showing an example of the slot management methodusing the scene base layout template of a photo album of type-K. Slotsother than the slot 1203 are not illustrated in FIG. 12B for thedescriptive convenience because it focuses on the slot management methodbased on FIG. 12A. Referring to FIG. 12B, a point O is the referencepoint of the photo album. Using the upper left corner of the doublespread of the photo album as the reference point, the application 80defines the position of each slot and manages/controls the layout of animage in each slot. The application 80 defines the rightward directionfrom the point as a positive (+) abscissa, the leftward direction as anegative (−) abscissa, the downward direction as a positive (+)ordinate, and the upward direction as a negative (−) ordinate. Thecoordinates of each point are expressed as (abscissa, ordinate). Forexample, the coordinates of the point O are (0, 0) [unit: cm].

A point W is the reference point of the slot 1203, which matches theupper left corner of the slot. In this way, the reference point of eachslot is set at the upper left corner of the slot. The point W is located1 cm to the right and 1 cm to below the point O and has coordinates(1, 1) [unit: cm]. L is the width of the slot 1203, which is 16 cm. H isthe height of the slot 1203, which is 6 cm. Hence, the points of therectangle of the slot 1203 and their coordinates are a point X (17, 1)[unit: cm], a point Y (17, 7) [unit: cm], and a point Z (1, 7) [unit:cm]. A point P is the center of the slot 1203, whose coordinates are (9,4) [unit: cm]. R is the rotation angle of the slot 1203. Since the slot1203 is not rotated in the example of FIG. 12B, the rotation angle is0°. In this way, the application 80 defines each slot laid out on thelayout template using coordinates and a rotation angle, and uses theinformation when managing/controlling the slots. The pieces of definedinformation are managed like a table shown in FIG. 15.

FIGS. 13A to 13E are views showing examples of the relationship oflayout between slots and still images. FIGS. 13A and 13B each show therelationship of layout between slots and still images on the template1102. FIGS. 13C and 13D each show the relationship of layout betweenslots and still images on the template 1141. FIG. 13E shows therelationship of layout between slots and still images on the template1200. In FIGS. 13A to 13E, Album Type is type of a photo album, Group isa region (group) number, Page is a page number, Slot is a slot number,and File is a file name. Referring to FIGS. 13A and 13B, slots of slotnumbers 1 to 4 correspond to the slots 1107 to 1110, respectively.Referring to FIGS. 13C and 13D, slot numbers 1 to 5 correspond to theslots 1143 to 1147, respectively. Referring to FIG. 13E, slot numbers 1to 7 correspond to the slots 1203 to 1209, respectively. In FIGS. 13A to13E, static*.jpg is a general still image such as a photo shot by adigital still camera or the like and saved, and dynamic*.jpg (* is anarbitrary number) is a still image extracted from a moving image.

FIG. 14 is a view showing an example of the scores of still images. InFIG. 14, File is a file name, Elements are elements that defineappropriateness (quality) as an image for a photo album, and Score isthe general score of the elements. The score is an index that is derivedfrom the points of the elements of Elements and indicates generalappropriateness from the viewpoint of layout on a template.

Elements include Color, Contrast, Brightness, Focus, and Composition.Each element represents appropriateness as an image for a photo album by0 (minimum) to 10 (maximum) points. The point of each element can beadded based on a predetermined reference. For example, when thedifference between the maximum value and the minimum value of brightnessvalues in a still image corresponds to a predetermined level, acorresponding point may be added as a contrast point. For example, whena triangular composition based on an object in a still image correspondsto a predetermined level, a corresponding point may be added as acomposition point. For example, when the sharpness of a change in thebrightness distribution in a still image corresponds to a predeterminedlevel, a corresponding point may be added as a focus point.

Original is an original score which is the average value of the pointsof Color, Contrast, Brightness, Focus, and Composition. Modified is ascore obtained by modifying the original score. To express a weightrepresenting the merit of a still image extracted from a moving image asa score, the application 80 increases the score by, for example,multiplying the original score by 1.2.

As for the merit, for example, the possibility that a still imageextracted from a moving image is specified as a layout target for atemplate can be raised by making the original score value large. Thatis, as for still images extracted from a moving image, still images ofhigher quality can be acquired by re-executing extraction. Hence, thequality of a finally completed template can be improved.

When laying out a still image in each slot, the application 80 employs ahigher one of the original score and the modified score and uses it tospecify an image more appropriate as a layout target for the template.Note that in the above-described example, the original score of eachstill image extracted from a moving image is multiplied by 1.2, therebyincreasing the score. The score of a still image extracted from a movingimage is thus controlled to be relatively higher than the score of ageneral still image such as a photo shot by a digital still camera orthe like and saved. However, the control method is not limited to this.For example, the score of the general still image such as a photo shotby a digital still camera or the like and saved may be multiplied by 0.8to decrease the score. The score of the still image extracted from themoving image may thus be controlled to be relatively higher than thescore of the general still image such as a photo shot by a digital stillcamera or the like and saved. In FIG. 14, “−” represents absence ofdata.

FIG. 15 is a view showing an example of the positions of the slots.Referring to FIG. 15, Slot is the slot number, Reference Pointrepresenting the reference point of a slot Length is the width of theslot, Height is the height of the slot, and Rotation is the rotationangle of the slot. FIG. 15 shows an example of the positions of theslots 1203 to 1209 of the layout template 1200 shown in FIG. 12A.

FIGS. 17A and 17B are views showing examples of association betweenstill images and moving images and classification of still images. FIG.17A shows an example according to this embodiment, and FIG. 17B shows anexample according to the second embodiment to be described later. InFIGS. 17A and 17B, JPEG is the file name of a still image, MPEG is thefile name of an MPEG moving image, Date & Time is the shootingdate/time, Info is information about classification, and Group is aclassification result. Frame is a frame number of a moving image,Related File is the file name of a file (moving image file/still imagefile) associated with a still image file or a moving image file. A stillimage file described in a JPEG cell is a still image file that isextracted from a moving image file described in the MPEG column of thesame row and saved. The format of Date & Time is year/month/day and time( 1/1000 sec). For example, “20140330100010334” described in the Date &Time cell of the row of dynamic15.jpg in FIG. 17A represents Mar. 30,2014, 10:00 and 334/1000 sec. In FIGS. 17A and 17B, “−” representsabsence of data.

FIG. 5 is a flowchart showing processing of automatically creating aphoto album. A program according to the flowchart of FIG. 5 is storedin, for example, the HDD 202, read out to the RAM 201, and executed bythe CPU 204. When the user selects the auto selection button 407 shownin FIG. 4A and presses the photo album creation button 412, theapplication 80 executes the processing from step S501 in accordance withthe flowchart shown in FIG. 5. In step S502, the application 80 selectsa folder (not shown) in the PC 1, which stores still image files ormoving image files, or still image files or moving image files in the PC1, and designates the folder or files as the storage destination of animage group or an image group necessary when creating a photo album.Note that in step S502, the user may be caused to select a folder in thePC 1, which stores still images or moving images, or still image filesor moving image files in the PC 1.

In step S503, the application 80 executes processing, shown in FIG. 16A,of acquiring a still image from a moving image file and saving it. Instep S504, for the still image saved in step S503, the application 80saves the position information (frame number) of a frame in the movingimage, which is the base of the still image, in association with thestill image, as shown in, for example, FIG. 17A.

In step S505, concerning each still image in the image group necessarywhen creating a photo album, the application 80 analyzes each elementshown in FIG. 14 which represents appropriateness as an image for aphoto album. A score is calculated and added to each still image as anevaluation value representing the appropriateness of each element.Images to which scores are added here are the still images in the stillimage file and the still images saved in step S503. Note that in stepS505, scores are added to both still images in the still image file andstill images extracted from the moving image file. In step S505, theapplication 80 also adds a score to each still image extracted from themoving image, which is added in step S503, in accordance with theappropriateness of each element as an image for a photo album.

In step S506, the application 80 modifies the score of each still imageextracted from the moving image. In step S506, to express a weightrepresenting the merit of each still image extracted from the movingimage as a score, the application 80 increases the score by, forexample, multiplying the original score by 1.2. When laying out a stillimage in each slot, the application 80 employs a higher one of theoriginal score and the modified score shown in FIG. 14. The application80 then lays out, in the slots, still images as many as the slots in thetemplate.

When the application 80 executes processing shown in FIG. 16B for eachstill image extracted from the moving image by executing the processingshown in FIG. 16A, still images more appropriate as images for a photoalbum can be extracted. Hence, the application 80 increases the score bymultiplying the original score of each still image extracted from themoving image by 1.2 as a weight corresponding to the merit.

The processing of laying out an image in each slot and creating a photoalbum will be described below using the layout template 1100 shown inFIG. 11A as an example. In step S507, the application 80 lays out stillimages in the slots of layout templates for a photo album of type-K,including the layout template 1100, in descending order of score.

In step S508, the application 80 determines whether a still imageextracted from the moving image is laid out in a slot. Upon determining,in step S509, that a still image extracted from the moving image is laidout in at least one of the slots, the application 80 advances to stepS510. Upon determining that a still image extracted from the movingimage is not laid out in any slot, the application 80 advances to stepS512 to end the processing shown in FIG. 5.

Assume that in step S507, static1.jpg, static2.jpg, static3.jpg, anddynamic20.jpg as a still image extracted from a moving image are laidout in the slots 1107 to 1110, as shown in FIG. 13A, based on the scoresshown in FIG. 14. In step S510, the application 80 executes processing,shown in FIG. 16B, of acquiring a still image more appropriate as alayout target on the template from the moving image and saving the stillimage based on dynamic20.jpg laid out in the slot 1110. Based onposition information (frame number) associated with the still imagedynamic20.jpg, all frames included within a predetermined range of apredetermined time (for example, a total of 2 sec including 1 precedingsec and 1 succeeding sec) before and after that position are analyzed,and a frame more appropriate for a photo album is extracted.

At this time, to extract a more appropriate frame, the application 80performs scoring of each frame image using the elements shown in FIG. 14and employs a frame of a higher original score. The application 80extracts a frame more appropriate for a photo album and saves theextracted frame as a still image dynamic24.jpg.

In step S511, the application 80 removes, from the slot 1110, the stillimage dynamic20.jpg laid out in the slot 1110. Instead, the application80 lays out the still image dynamic24.jpg more appropriate for a photoalbum, which is saved in step S510, in the slot 1110. As a result, theimages shown in FIG. 13B are laid out in the slots 1107, 1108, 1109, and1110. In step S512, the application 80 ends the processing ofautomatically creating a photo album and displays the preview/editingscreen 415 shown in FIG. 4B.

As described above, it is possible to provide a photo album creationapplication that handles both a still image and a moving image togetherand automatically extracts and lays out a more appropriate image withouta cumbersome operation of the user.

In the above-described example, in step S506, the original score of thestill image extracted from the moving image is multiplied by 1.2,thereby increasing the score. Controlling to make the score of the stillimage extracted from the moving image relatively higher than the scoreof a general still image such as a photo shot by a digital still cameraor the like and saved is thus implemented. However, the presentinvention is not limited to this control method. For example, theapplication 80 may multiply the score of a general still image such as aphoto shot by a digital still camera or the like and saved by 0.8 todecrease the score. The score of the still image extracted from themoving image may be controlled to be relatively higher than the score ofthe general still image such as a photo shot by a digital still cameraor the like and saved.

In the above-described example, in step S506, the original score of thestill image extracted from the moving image is multiplied by 1.2 toincrease the score, or the score of a general still image such as aphoto shot by a digital still camera or the like and saved is multipliedby 0.8 to decrease the score. However, the present invention is notlimited to these examples. For example, upon determining that qualitybased on the resolution and the like of the moving image is high, theoriginal score of a still image extracted from the moving image may bemultiplied by 1.5. In this case, if the resolution is equal to or morethan a reference value, the quality may be determined to be high. Inaddition, if quality based on the resolution and the like of a generalstill image such as a photo shot by a digital still camera or the likeand saved is low, the original score of the still image may bemultiplied by 0.5. With this arrangement, a score value representingappropriateness as an image for a photo album can flexibly becontrolled.

In the above-described example, after a score is added to a still imagein step S505, the score is modified in step S506. However, the presentinvention is not limited to this. In step S505, processing may beperformed to add a high score to a still image extracted from a movingimage. For example, a first evaluation criterion to readily add a highscore and a second evaluation criterion to readily add a score lowerthan that of the first evaluation criterion are stored in the HDD 202.In step S505, the CPU 204 may add a score to the still image extractedfrom the moving image based on the first evaluation criterion and add ascore to each still image other than the still image based on the secondevaluation criterion.

Note that in the above-described example, in step S511, the still imagedynamic20.jpg laid out in the slot 1110 is removed from the slot 1110,and the still image dynamic24.jpg more appropriate for a photo album,which is saved in step S510, is laid out in the slot 1110 instead.However, the present invention is not limited to this control method.For example, by the original score of the still image dynamic24.jpg, therelationship of layout between slots and still images on the template1102 shown in FIG. 13B may be reviewed. For example, assume that theoriginal score of dynamic24.jpg is 7.5. In this case, the score(=original score) of dynamic24.jpg 24 is higher than the original score(7.0) of static3.jpg. Hence, the application 80 lays out dynamic24.jpgin slot 3 shown in FIG. 13B and static3.jpg in slot 4 shown in FIG. 13B.

FIGS. 16A and 16B are flowcharts showing processing of acquiring a stillimage from a moving image and saving the still image. FIGS. 16A and 16Billustrate details of processing in step S503. Programs according to theflowcharts of FIGS. 16A and 16B are stored in, for example, the HDD 202,read out to the RAM 201, and executed by the CPU 204. FIG. 16A is aflowchart showing an example of processing of acquiring a still imagefrom a moving image and saving the still image. FIG. 16B is a flowchartshowing processing of acquiring a more appropriate still image from amoving image and saving the still image.

In step S503 of FIG. 5, the application 80 executes the processing ofacquiring a still image from a moving image and saving the still image.Then, the application 80 starts processing from step S1611 of FIG. 16A.In step S1612, the application 80 searches for a still image or a movingimage stored in the folder designated in step S502 of FIG. 5 or a movingimage from the image group designated in step S502 of FIG. 5.

In step S1613, the application 80 extracts a plurality of frames fromthe found moving image at a time interval designated in the frameextraction time interval designation portion 413 shown in FIG. 4A. Thetime interval designated in the frame extraction time intervaldesignation portion 413 is 1 sec that is the default value. In stepS1614, the application 80 converts each extracted frame into a stillimage (JPEG file), and saves the still images. Note that at this time,the still image need not always be saved as a file. Instead, the CPU 204may store the still image in the RAM 201 in, for example, the bitmapformat. At this time, as shown in FIG. 17A, the application 80 storeseach still image in association with the file name of the moving imagefrom which the still image has been extracted and the positioninformation (frame number) of a frame as the base of the still image.The information of the file of the moving image from which the stillimage has been extracted includes the full path of the storagedestination. However, FIG. 17A does not illustrate the full path of thestorage destination.

In step S1615, the application 80 adds these still images to the imagegroup necessary when creating a photo album. In step S1616, theapplication 80 ends the processing shown in FIG. 16A and returns to theinvocation source of the processing.

Assume that in step S507, the still image files static1.jpg,static2.jpg, static3.jpg, and dynamic20.jpg are laid out in the slots1107 to 1110, as shown in FIG. 13A, based on the scores shown in FIG.14. Here, the still image file dynamic20.jpg is a file saved based on aframe extracted from a moving image file video1.mp4, as shown in FIG.17A. Note that video1.mp4 is a moving image file of MPEG4 format, andits frame rate is 30 fps (frames/sec).

When processing of acquiring a more appropriate still image from themoving image and saving the still image is executed in step S510 of FIG.5, the application 80 starts the processing shown in FIG. 16B from stepS1621 based on the still image dynamic20.jpg extracted from the movingimage and laid out in the slot 1110. The processing shown in FIG. 16B isprocessing of acquiring a still image more appropriate as a layouttarget for the template from the moving image and saving the stillimage. In step S1622, the application 80 acquires position information(frame number) associated with the still image dynamic20.jpg. In stepS1623, based on the position information (frame number), the application80 extracts all frames included within a predetermined range of apredetermined time (for example, a total of 2 sec including 1 precedingsec and 1 succeeding sec) before and after that position.

In step S1624, the application 80 analyzes all the extracted frames andselects a frame more appropriate for a photo album. At this time, toselect a more appropriate frame, the application 80 performs storing ofeach frame image using the elements shown in FIG. 14 and selects a frameof a higher original score. For example, the frame of the highestoriginal score is selected out of the extracted frames.

In step S1625, the application 80 converts the selected frame into astill image (JPEG file), and saves the still image. Here, theapplication 80 selects a frame more appropriate for a photo album basedon the still image dynamic20.jpg, and saves the selected frame as astill image dynamic24.jpg. In step S1627, the application 80 ends theprocessing shown in FIG. 16B and returns to the invocation source of theprocessing.

According to the above-described processing, when laying out stillimages on a template, a high score is readily added to a still imageextracted from a moving image, and the still image is readily laid outon the template.

For example, when a scene is shot by a camera as a still image, the shotimage is acquired at the timing at which the user presses the shutterbutton. Hence, if the image is out of focus at that timing, the imageblurs. On the other hand, in a moving image, a number of images (stillimages) are shot for one scene. Hence, an appropriate image with littleblur may be included in the number of images.

According to the above-described processing, high scores arepreferentially added to a plurality of still images extracted from amoving image. Out of the plurality of still images, images of highevaluation are laid out. It is therefore possible to lay out appropriateimages with little blur.

Second Embodiment

FIG. 18 is a view showing examples of frames of a moving image. FIG. 18shows an example of a moving image file video444.mp4. Referring to FIG.18, a number added under each frame is the frame number of the frame.Dynamic44.jpg, dynamic45.jpg, and dynamic46.jpg are still image filesthat an application 80 has extracted from frames 3, 4, and 5 ofvideo444.mp4, respectively, and saved. Dynamic54.jpg, dynamic55.jpg, anddynamic56.jpg are still image files that the application 80 hasextracted from frames 925, 926, and 927 of video444.mp4, respectively,and saved. Dynamic87.jpg, dynamic88.jpg, and dynamic89.jpg are stillimage files that the application 80 has extracted from frames 3334,3335, and 3336 of video444.mp4, respectively, and saved. As shown inFIG. 18, frames 1 to 5, frames 924 to 928, and frames 3333 to 3337 areframe groups of different scenes.

For a scene, still images are classified based on an object included inthe still images. For example, if the type of an object included instill images changes, the scene is discriminated as a different scene.Scene discrimination may be done based on another criterion. Forexample, scenes may be discriminated by the same or similar composition.

FIG. 6 is a flowchart showing processing of automatically creating aphoto album. A program according to the flowchart of FIG. 6 is storedin, for example, an HDD 202, read out to a RAM 201, and executed by aCPU 204. The user selects one of a time selection button 408, a sceneselection button 409, and a theme selection button 410 shown in FIG. 4A,and presses a photo album creation button 412. In response to thepressing, the application 80 executes the processing from step S601 inaccordance with the flowchart shown in FIG. 6. In step S602, theapplication 80 selects a folder (not shown) in a PC 1, which storesstill images or moving images, or still images or moving images in thePC 1, and designates the folder or files as the storage destination ofan image group or an image group necessary when creating a photo album.Note that in step S602, the user may be caused to select a folder in thePC 1, which stores still images or moving images, or still images ormoving images in the PC 1.

In step S603, the application 80 executes still image classificationprocessing shown in FIG. 7. In step S604, according to a classificationmethod (time base, scene base, or theme base) in the still imageclassification processing of step S603, the application 80 executesprocessing shown in FIG. 8A, 8B, or 8C corresponding to the method. Eachof the processes shown in FIGS. 8A, 8B, and 8C is processing ofextracting a still image from a moving image in correspondence withclassification. Note that as for the classification method, thisembodiment includes not only a case in which one of the time base, thescene base, and the theme base is applied but also a case in which someof the methods are applied in combination.

In step S605, based on the classification method applied in step S603,the application 80 selects a layout template associated with theclassification method. Assume that the application 80 selects the themebase classification method in step S603. In step S605, the application80 selects a layout template associated with the theme baseclassification method. Assume that the application 80 selects a layouttemplate including a layout template 1140 shown in FIG. 11C.

In step S606, the application 80 lays out still images associated witheach other in slots within a series of regions associated in the layouttemplate. At this time, if the number of still images classified by theclassification (for example, theme) is larger than a predeterminednumber of slots in the series of regions, scoring as described in thefirst embodiment may be performed, and the still images may be assignedto the predetermined number of slots in descending order of score.

In step S607, the application 80 confirms the still images laid out inthe slots, and determines whether a plurality of still images extractedfrom a moving image are laid out. Upon determining in step S608 that aplurality of still images are laid out, the application 80 advances tostep S609. Upon determining that a plurality of still images are notlaid out, the application 80 advances to step S615 to end the processingshown in FIG. 6.

Assume that still image files static14.jpg, static28.jpg, dynamic44.jpg,dynamic45.jpg, and dynamic46.jpg are laid out in slots 1143 to 1147, asshown in FIG. 13C. In step S609, the application 80 determines whetherimages of the same scene are included in the plurality of still imagesextracted from a moving image and laid out. Upon determining in stepS610 that images of the same scene are included, the application 80advances to step S611. Upon determining that images of the same sceneare not included, the application 80 advances to step S615 to end theprocessing shown in FIG. 6.

In step S611, the application 80 determines the presence/absence ofanother still image of a different scene out of the still imagesextracted from the moving image. Upon determining in step S612 thatanother still image of a different scene exists, the application 80advances to step S613. Upon determining that another still image of adifferent scene does not exist, the process advances to step S614.

In step S613, the application 80 replaces one of the still images of thesame scene laid out in the slots out of the still images extracted fromthe moving image with the other still image of the different sceneextracted from the moving image, and lays out the still image of thedifferent scene in the slot. As a result, the still image dynamic45.jpglaid out in the slot 1146 is replaced with a still image dynamic55.jpg,and the still image dynamic46.jpg laid out in the slot 1147 is replacedwith a still image dynamic88.jpg. As shown in FIG. 13D, still imagefiles static14.jpg, static28.jpg, dynamic44.jpg, dynamic55.jpg, anddynamic88.jpg are laid out in the slots 1143 to 1147, respectively.

In step S614, the application 80 replaces one of the still images of thesame scene laid out in the slots out of the still images extracted fromthe moving image with a still image of the same classification, which isnot a still image extracted from the moving image, and lays out thestill image in the slot. At this time, the replacement may be done suchthat the score value shown in FIG. 14 becomes large. Alternatively, thereplacement may be done such that the point of one of the elements shownin FIG. 14 becomes large. As a result, the quality of a finallycompleted template can be improved. In step S615, the application 80ends the processing of automatically creating a photo album shown inFIG. 6 and displays a preview/editing screen 415 shown in FIG. 4B.

FIG. 7 is a flowchart showing still image classification processing. Aprogram according to the flowchart of FIG. 7 is stored in, for example,the HDD 202, read out to the RAM 201, and executed by the CPU 204. Whenthe application 80 executes still image classification processing instep S603 of FIG. 6, the application 80 starts the still imageclassification processing from step S701 of FIG. 7.

In step S702, the application 80 acquires the selection state of amoving image file processing method selection portion 406 by the user,and determines the classification method based on the processing methodof extracting a still image from a moving image, which is selected bythe user. Upon determining in step S703 that the classification methodis the time base classification method, the application 80 advances tostep S704. Upon determining that the classification method is not thetime base classification method, the application 80 advances to stepS705.

In step S704, the application 80 refers to the Date Time information ofExif information of a still image (JPEG file), the creation date,updating date/time, and access date/time of the still image file, andthe like, and classifies the still image based on a shooting date/timeobtained from these pieces of information. The classificationinformation and the classification result (group) are saved inassociation with the target still image. At this time, if the Date Timeinformation of Exif information does not exist, the application 80 savesa shooting date/time obtained from the creation date, updatingdate/time, and access date/time of the file, and the like in associationwith the still image.

As the classification method, for example, a method of putting stillimages of the same day together into the same classification or puttingstill images of the same time information, for example, the same monthtogether into the same classification is usable. For example, the row ofstatic102.jpg shown in FIG. 17B indicates a case in which theclassification is done by this method. Information representing that thestill image is classified by a time between 20:00 and 21:00 (20:00 to20:59) in Dec. 24, 2013 is stored in Info (classification information)in association with static102.jpg.

Upon determining in step S705 that the classification method is thescene base classification method, the application 80 advances to stepS706. Upon determining that the classification method is not the scenebase classification method, the application 80 advances to step S707.

In step S706, the application 80 analyzes the color information,histogram, and the like of a still image, classifies the still imagebased on the analysis result, and saves the classification informationand the classification result (group) in association with the targetstill image. At this time, the application 80 also saves the colorinformation and the histogram information in association with the stillimage. As the classification method, for example, a method of puttingstill images of similar tints together into the same classification orputting still images whose RGB or YCrCb histograms are coincident orwhose chromaticities fall within a predetermined threshold rangetogether into the same classification is usable. For example, the row ofstatic120.jpg shown in FIG. 17B indicates a case in which theclassification is done by this method. Information representing that thestill image is classified by histogram 4 (histogram4-*, * is a naturalnumber representing a histogram value determined as a group of histogram4) is stored in Info (classification information) in association withstatic120.jpg.

Upon determining in step S707 that the classification method is thetheme base classification method, the application 80 advances to stepS708. Upon determining that the classification method is not the themebase classification method, the application 80 advances to step S709.

In step S708, the application 80 analyzes an object included in a fileselected by the user using a file selection button 411, and classifiesthe still image based on the analysis result. The classificationinformation and the classification result (group) are saved inassociation with the target still image. At this time, the application80 also saves the object information in association with the stillimage. Examples of objects are a pet such as a dog and a cat, a flower,a train, a car, a motorcycle, and the like. Note that a human (a person,a face, or the like) is also included in examples of objects. Forexample, the row of static155.jpg shown in FIG. 17B indicates a case inwhich the classification is done by this method. Informationrepresenting that the still image is classified by object 8(object8-*, * is a natural number representing information that definesan object determined as a group of object 8) is stored in Info(classification information) in association with static155.jpg.

In step S709, the application 80 classifies the still image based onanother method, and saves the result in association with the targetstill image. At this time, the application 80 also saves the informationof the other method in association with the still image. Note that whenthe user operates a main screen 401 according to this embodiment, theapplication 80 does not execute the processing of step S709 in somecases. In step S710, the application 80 ends the still imageclassification processing shown in FIG. 7 and returns to the invocationsource of the processing.

FIGS. 8A to 8C are flowcharts showing processing of extracting a stillimage from a moving image. A program according to each of the flowchartsof FIGS. 8A to 8C is stored in, for example, the HDD 202, read out tothe RAM 201, and executed by the CPU 204. FIG. 8A is a flowchart showingprocessing of extracting a still image from a moving image on the timebase. FIG. 8B is a flowchart showing processing of extracting a stillimage from a moving image on the scene base. FIG. 8C is a flowchartshowing processing of extracting a still image from a moving image onthe theme base.

Assume that the user selects the time selection button 408 shown in FIG.4A, and presses the photo album creation button 412. In this case, whenthe application 80 executes processing of extracting a still image froma moving image in step S604 of FIG. 6, the application 80 starts theprocessing of extracting a still image from a moving image in step S811of FIG. 8A.

In step S812, based on classification information associated with astill image in step S704 and the Date Time information of Exifinformation of the still image or a shooting date/time associated withthe still image, the application 80 extracts an associated frame from amoving image. At this time, the application 80 acquires the shootingdate/time of the moving image first, and specifies a moving image with acoincident classification using the classification information (forexample, same day and same month) associated with the still imageobtained in step S704. The classification information is saved inassociation with the specified moving image. Next, the application 80acquires the Date Time information of Exif information or a shootingdate/time associated with the still image from a still image of the sameclassification, and extracts a frame with a coincident shootingdate/time from the moving image of the same classification. At thistime, using the shooting date/time as a reference time, the application80 analyzes all frames included within a predetermined range of apredetermined time (for example, a total of 6 sec including 3 precedingsec and 3 succeeding sec) before and after the shooting date/time, andextracts a frame more appropriate for a photo album.

The application 80 may extract a plurality of frames. At this time, toextract a more appropriate frame, the application 80 performs scoring ofeach frame image using the elements shown in FIG. 14 and employs theframes in descending order of original score. As for the number offrames to be employed, for example, the number of slots capable oflaying out still images extracted from a moving image may be determinedas the number of frames to be employed.

In step S813, the application 80 saves the frame extracted in step S812as a still image. Note that if a frame with a coincident shootingdate/time does not exist in step S812, a still image is extracted byexecuting the processing shown in FIG. 16A for the moving image.

In step S814, the application 80 saves the classification information,the classification result, and the associated file in association withthe still image extracted from the moving image. For example, the row ofdynamic301.jpg shown in FIG. 17B indicates this case. Informationrepresenting that the still image is classified by a time between 20:00and 21:00 (20:00 to 20:59) in Dec. 24, 2013 is stored in Info(classification information) in association with the still image. Inaddition, group information date Time 1 is stored in Group(classification result) in association with the still image.Furthermore, static102.jpg is stored in Related File (associated file)in association with the still image. In step S815, the application 80ends the processing shown in FIG. 8A and returns to the invocationsource of the processing.

Assume that the user selects the scene selection button 409 shown inFIG. 4A, and presses the photo album creation button 412. In this case,when the application 80 executes processing of extracting a still imagefrom a moving image in step S604 of FIG. 6, the application 80 startsthe processing of extracting a still image from a moving image in stepS821 of FIG. 8B.

In step S822, based on classification information such as colorinformation or histogram associated with a still image in step S706 ofFIG. 7, the application 80 extracts an associated frame from a movingimage. At this time, the application 80 extracts a frame from eachmoving image, analyses the color information, histogram, and the like ofthe frame, and compares them with the classification information (colorinformation, histogram, and the like) associated with the still imageobtained in step S706 of FIG. 7. As a result, if the pieces ofclassification information are coincident or similar, the classificationinformation is saved in association with the moving image. Theapplication 80 also saves the classification information in associationwith the frame with the coincident classification information. Theapplication 80 analyzes all frames with coincident classificationinformation, and extracts a frame appropriate for a photo album. Theapplication 80 may extract a plurality of frames. Extraction of anappropriate frame (the order of score and the number of frames to beemployed) is performed based on the same reference as in FIG. 8A. Instep S823, the application 80 saves the frame extracted in step S822 asa still image.

In step S824, the application 80 saves the classification information,the classification result, and the associated file in association withthe still image extracted from the moving image. For example, the row ofdynamic321.jpg shown in FIG. 17B indicates this case. Informationrepresenting that the still image is classified by histogram 4(histogram4-*, * is a natural number representing a histogram valuedetermined as a group of histogram 4) is stored in Info (classificationinformation) in association with the still image. In addition, groupinformation histogram 4 is stored in Group (classification result) inassociation with the still image. Furthermore, static120.jpg is storedin Related File (associated file) in association with the still image.In step S825, the application 80 ends the processing shown in FIG. 8Band returns to the invocation source of the processing.

Assume that the user selects the theme selection button 410 shown inFIG. 4A, and presses the photo album creation button 412. In this case,when the application 80 executes processing of extracting a still imagefrom a moving image in step S604 of FIG. 6, the application 80 startsthe processing of extracting a still image from a moving image in stepS831 of FIG. 8C.

In step S832, based on the classification information of an objectassociated with a still image in step S708 of FIG. 7, the application 80extracts an associated frame from a moving image. At this time, theapplication 80 extracts a frame from each moving image, analyses anobject included in the frame, and compares it with the classificationinformation (object) associated with the still image obtained in stepS708 of FIG. 7. As the result of comparison, if the pieces ofclassification information are coincident or similar, the classificationinformation is saved in association with the moving image. Theapplication 80 also saves the classification information in associationwith the frame with the coincident classification information. Theapplication 80 analyzes all frames with coincident classificationinformation, and extracts a frame appropriate for a photo album. Theapplication 80 may extract a plurality of frames. Extraction of anappropriate frame (the order of score and the number of frames to beemployed) is performed based on the same reference as in FIG. 8A.

In step S833, the application 80 saves the frame extracted in step S832as a still image. In step S834, the application 80 saves theclassification information, the classification result, and theassociated file in association with the still image extracted from themoving image. For example, the row of dynamic341.jpg shown in FIG. 17Bindicates this case. Information representing that the still image isclassified by object 8 (object8-*, * is a natural number representinginformation that defines an object determined as a group of object 8) isstored in Info (classification information) in association with thestill image. In addition, group information object 8 is stored in Group(classification result) in association with the still image.Furthermore, static155.jpg is stored in Related File (associated file)in association with the still image. In step S835, the application 80ends the processing shown in FIG. 8C and returns to the invocationsource of the processing.

As described above, based on a classified still image, a still imageassociated with the still image is automatically extracted from a movingimage, and the still images are laid out in a series of regions in alayout. This makes it possible to provide a photo album creationapplication that automatically extracts appropriate associated imagesfrom a moving image and lays out them without a cumbersome operation ofthe user. It is also possible to provide a photo album creationapplication that extracts a plurality of still images of differentscenes from one moving image and lays out them in a series of regions,thereby automatically extracting a variety of images and laying out themwithout a cumbersome operation of the user.

Third Embodiment

FIG. 9 is a flowchart showing processing of automatically creating aphoto album. A program according to the flowchart of FIG. 9 is storedin, for example, an HDD 202, read out to a RAM 201, and executed by aCPU 204. The user selects one of a time selection button 408, a sceneselection button 409, and a theme selection button 410 shown in FIG. 4A,and presses a photo album creation button 412. In response to thepressing, an application 80 executes the processing from step S901 inaccordance with the flowchart shown in FIG. 9.

In step S902, the application 80 selects a folder (not shown) in a PC 1,which stores still images or moving images, or still images or movingimages in the PC 1, and designates the folder or files as the storagedestination of an image group or an image group necessary when creatinga photo album. Note that in step S902, the user may be caused to selecta folder in the PC 1, which stores still images or moving images, orstill images or moving images in the PC 1.

In step S903, the application 80 executes still image classificationprocessing shown in FIG. 7 as described in the second embodiment. Here,static11.jpg and static12.jpg are photos (still images) shot by adigital camera and saved. Classification information associated withstatic11.jpg is coincident with classification information associatedwith static12.jpg.

In step S904, according to a classification method (time base, scenebase, or theme base) in the still image classification processing ofstep S903, the application 80 executes processing shown in FIG. 8A, 8B,or 8C corresponding to the method. Each of the processes shown in FIGS.8A, 8B, and 8C is processing of extracting a still image from a movingimage. Note that as for the classification method, this embodimentincludes not only a case in which one of the time base, the scene base,and the theme base is applied but also a case in which some of themethods are applied in combination. Here, dynamic22.jpg, dynamic43.jpg,dynamic50.jpg, dynamic78.jpg, and dynamic85.jpg are still imagesextracted from one moving image. All pieces of classificationinformation associated with these still images are coincident with eachother, and are also coincident with the classification informationassociated with static11.jpg and the classification informationassociated with static12.jpg.

In step S905, based on the classification method applied in step S903,the application 80 selects a layout template associated with theclassification method. Assume that the application 80 selects the scenebase classification method in step S903. In step S905, the application80 selects a layout template associated with the scene baseclassification method.

Assume that the application 80 selects a layout template including alayout template 1200 shown in FIG. 12A. In step S906, the application 80lays out general still images associated with each other in slots eachsuitable for a general still image such as a photo shot by a digitalstill camera or the like and saved within a series of regions associatedin the layout template. Here, general still images static11.jpg andstatic12.jpg associated with each other are laid out in slots 1203 and1204. At this time, if the number of classified still images is largerthan a predetermined number of slots in the series of regions, scoringas described in the first embodiment may be performed, and the stillimages may be assigned to the predetermined number of slots indescending order of score.

In step S907, the application 80 lays out the still images associatedwith each other and extracted from the moving image, in the slotssuitable for still images extracted from a moving image in the series ofregions associated in the layout template. Here, dynamic22.jpg,dynamic43.jpg, dynamic50.jpg, dynamic78.jpg, and dynamic85.jpgassociated with each other and extracted from the moving image are laidout in slots 1205 to 1209.

In step S908, the application 80 ends the processing of automaticallycreating a photo album shown in FIG. 9 and displays a preview/editingscreen 415 shown in FIG. 4B.

As described above, the still images each having a high resolution arelaid out in the slots 1203 and 1204, each of which is somewhat large andsuitable for a general still image such as a photo shot by a digitalstill camera or the like and saved, within the series of regionsassociated in the layout template. In addition, the still imagesextracted from one moving image are laid out in the series of continuousslots 1205 to 1209, each of which is suitable for a still imageextracted from a moving image, within the series of regions. As aresult, it is possible to provide a photo album creation applicationthat automatically lays out images with realism by making use of thequality of still images and the motion of moving images without acumbersome operation of the user.

Other Embodiments

The present invention can be implemented even when a program thatimplements at least one function of the above-described embodiments issupplied to a system or an apparatus via a network or a storage medium,and at least one processor in the computer of the system or apparatusreads out the program and executes processing. The present invention canalso be implemented by a circuit (for example, ASIC) that implements atleast one function.

In the above-described embodiments, the photo album creation application80 as shown in FIGS. 4A and 4B has been described as an example of theapplication. However, the application is not limited to this example,and an arbitrary application having the same image extraction functioncan also be implemented.

In the above-described embodiments, a personal computer is assumed asthe information processing apparatus. However, an arbitrary informationprocessing apparatus (terminal) which enables the same use method, suchas a portable telephone, a portable information terminal, a digitalstill camera, a digital video camera, a portable music player, a gamemachine, a set-top box, or an Internet appliance, may be used.

In the above-described embodiments, for example, an OS equivalent toWindows® 8 is used as the OS. However, the OS is not limited to this,and an arbitrary OS can be used. In the above-described embodiments,Ethernet® is used as a configuration example of the network 4. However,the network configuration is not limited to this, and an arbitrarynetwork, for example, wireless LAN, IEEE 1394, or Bluetooth® may beused.

As described above, according to the first embodiment, it is possible toprovide a photo album creation application that handles both a stillimage and a moving image together and automatically extracts and laysout a more appropriate image without a cumbersome operation of the user.

According to the second embodiment, based on a classified still image, astill image associated with the still image is automatically extractedfrom a moving image, and the still images are laid out in a series ofregions. As a result, it is possible to provide a photo album creationapplication that automatically extracts appropriate associated imagesand lays out them without a cumbersome operation of the user. It is alsopossible to provide a photo album creation application that extracts aplurality of still images of different scenes from one moving image andlays out them in a series of regions, thereby automatically extracting avariety of images and laying out them without a cumbersome operation ofthe user.

According to the third embodiment, appropriate still images are laidout, in a layout template, in slots each suitable for a general stillimage such as a photo shot by a digital still camera or the like andsaved and in slots each suitable for a still image extracted from amoving image. As a result, it is possible to provide a photo albumcreation application that automatically lays out images with realism bymaking use of the quality of still images and the motion of movingimages without a cumbersome operation of the user.

As the processing method of classifying a still image and the processingmethod of extracting a still image from a moving image, examples using ashooting date/time, a color and histogram, and an object have beendescribed. However, the methods are not limited to these examples. Thepresent invention can be implemented using an arbitrary processingmethod capable of achieving the same object. A shooting date/time, acolor and histogram, an object, another processing method, and the likemay be combined to classify a still image or extract a still image froma moving image. When the plurality of processing methods are combined,the level of completion of a photo album can be improved.

As the image analysis methods, examples using the elements Color,Contrast, Brightness, Focus, and Composition have been described.However, the methods are not limited to these examples. The presentinvention can be implemented using an arbitrary analysis method capableof achieving the same object.

While the present invention has been described in detail with referenceto embodiments, the invention is not limited to the specificembodiments, and various changes and modifications can be made withinthe spirit and scope of the present invention.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-045090, filed Mar. 6, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A method, comprising: evaluating a first stillimage included in a still image file and a second still image extractedfrom a moving image included in a moving image file as layout targets ona template; specifying a still image of the layout target on thetemplate out of the first still image and the second still image inaccordance with the evaluation; and laying out the specified still imageof the layout target on the template, wherein the first still image andthe second still image are evaluated such that a possibility that thesecond still image is specified as the layout target on the templatebecomes higher than that for the first still image.
 2. The methodaccording to claim 1, wherein in the evaluating, evaluation values arecalculated using a common evaluation criterion for the first still imageand the second still image, and at least one of the evaluation valuecalculated for the first still image and the evaluation value calculatedfor the second still image is modified such that the possibility thatthe second still image is specified as the layout target on the templatebecomes high.
 3. The method according to claim 2, wherein in thespecifying, still images of the layout targets as many as the number ofstill images that can be laid out in the template are specified indescending order of modified evaluation value.
 4. The method accordingto claim 1, further comprising: in a case where the second still imageis specified, acquiring, from the moving image file, a third still imagewhose evaluation value is larger than the evaluation value of thespecified second still image; and replacing the second still image withthe acquired third still image in the laying out.
 5. The methodaccording to claim 4, wherein in the acquiring, the third still image isacquired from frames in a predetermined range before and after a framecorresponding to the specified second still image.
 6. The methodaccording to claim 1, further comprising: classifying the first stillimage; and extracting the second still image from the moving image filebased on a criterion of the classification.
 7. The method according toclaim 6, further comprising: in a case where the second still imageextracted based on the criterion of the classification comprises aplurality of second still images, determining whether the plurality ofsecond still images correspond to the same scene; upon determining thatthe plurality of second still images correspond to the same scene,extracting a fourth still image corresponding to a scene different fromthe scene from the moving image file in the extracting; and replacingthe second still image with the extracted fourth still image in thelaying out.
 8. The method according to claim 7, wherein the scene isdetermined by an object included in the second still image.
 9. Themethod according to claim 6, wherein the criterion of the classificationincludes at least one of color information, object information, and timeinformation.
 10. The method according to claim 1, wherein the templateincludes a slot capable of laying out a still image, and in the layingout, the specified still image of the layout target is laid out on thetemplate in accordance with an attribute of the slot.
 11. The methodaccording to claim 10, wherein the attribute of the slot is a type ofthe still image.
 12. The method according to claim 1, wherein in theevaluating, the evaluation is done based on at least one of a color,contrast, brightness, focus, and composition in an image.
 13. A methodcomprising: classifying a first still image included in a still imagefile; extracting a second still image from a moving image file based ona criterion of the classification; and laying out the first still imageand the extracted second still image on a template.
 14. The methodaccording to claim 13, further comprising: in a case where the secondstill image extracted based on the criterion of the classificationcomprises a plurality of second still images, determining whether theplurality of second still images correspond to the same scene; upondetermining that the plurality of second still images correspond to thesame scene, extracting a third still image corresponding to a scenedifferent from the scene from the moving image file; and replacing thesecond still image with the acquired third still image in the layingout.
 15. The method according to claim 14, wherein the scene isdetermined by an object included in the second still image.
 16. Themethod according to claim 13, wherein the criterion of theclassification includes at least one of color information, objectinformation, and time information.
 17. An information processingapparatus comprising: an evaluation unit configured to evaluate a firststill image included in a still image file and a second still imageextracted from a moving image included in a moving image file as layouttargets on a template; a specifying unit configured to specify a stillimage of the layout target on the template out of the first still imageand the second still image in accordance with the evaluation by theevaluation unit; and a layout unit configured to lay out the still imageof the layout target specified by the specifying unit on the template,wherein the evaluation unit evaluates the first still image and thesecond still image such that a possibility that the second still imageis specified as the layout target on the template becomes higher thanthat for the first still image.
 18. A non-transitory computer-readablestorage medium storing a program that causes a computer to function soas to: evaluate a first still image included in a still image file and asecond still image extracted from a moving image included in a movingimage file as layout targets on a template; specify a still image of thelayout target on the template out of the first still image and thesecond still image in accordance with the evaluation; and lay out thespecified still image of the layout target on the template, wherein whenperforming the evaluation, the first still image and the second stillimage are evaluated such that a possibility that the second still imageis specified as the layout target on the template becomes higher thanthat for the first still image.